Welcome back to deep learning and today we want to talk about the last part of

object detection and segmentation and we want to look into the concept of

instant segmentation. So let's have a look at our slides you see this is

already the last part part five and now we want to talk about instant

segmentation. So we not just want to detect where pixels with cubes are or

where pixels of cups are but we want to really figure out which pixels belong to

what cube. So this is essentially a combination of object detection and

semantic segmentation. Examples for potential applications are information

about occlusion, counting, the number of elements belonging to the same class,

detecting object boundaries for example of gripping objects and robotics this is

very important and there's examples in the literature simultaneous detection and

segmentation, deep mask, sharp mask and mask RCNN and reference 10. So let's look

at reference 10 in a little more detail. So we essentially go back to the start

we combine the object detection and the segmentation and we use RCNN for the

object detection and the object detection essentially solves the

instance separation and then the segmentation refines the bounding boxes

per instance. So the workflow is a two-stage procedure you have the region

proposal that proposes the object bounding boxes and then you have the

classification using a bounding box regression and the segmentation in

parallel. So you have a multitask loss that essentially combines the pixel-wise

classification loss so the segmentation loss, the box loss and the class loss for

producing the right class per bounding box. So you have these three terms that

are then combined in a multitask loss. So let's look in some more detail into the

two-stage procedure. You have two different options here for two-stage

networks you can have a joint branch that is working on the ROIs and then

splits at a later stage into the segmentation of the mask and the class

and bounding box prediction or you can split early and then run that into

separate networks. In both versions you have this multitask loss that combines

the pixel-wise segmentation loss, the box loss and the class loss. Let's have a

look at some examples and these are results again from mask RCNN and you

can see that to be honest these are quite impressive results. So there are

really difficult cases you identify where the persons are and you also show

that the different persons of course are different instances. So very impressive

results. So let's summarize what we've seen so far. The segmentation is

commonly solved by architectures analyzing the image and subsequently

refining the course results. Fully convolutional networks preserve the

spatial layout and enable arbitrary input sizes with pooling. We can use

object detectors and implement them as a sequence of region proposals and

classification then this leads essentially to the family of RCNN type

of networks. Alternatively you can go to single shot detectors and we looked at

YOLO which is a very common and very fast technique YOLO 9000 and we looked

into retina net if you really have a scale dependency and you want to detect

on many different scales like for the example of histological slice

processing. So object detection and segmentation are closely related and

combinations are common as you have seen here for the purpose of instant

segmentation. So let's look at what we still have to talk about in this lecture

and coming up very soon is methods to relieve the burden of labeling. So we

will talk about weekly annotation, how we can generate labels which then also

leads to the concept of self-supervision which is a very popular topic right now

and it's been very heavily used in order to generate better networks in order to